fix(dflash): derive n_target_layers fallback in gguf_draft_loader#138
Merged
davide221 merged 2 commits intoMay 12, 2026
Merged
Conversation
Follow-up to merged Luce-Org#79 ("read model params from GGUF at runtime, support any qwen35 size"). Luce-Org#79 covers the target loader and the common drafter fields, but the fallback chain in gguf_draft_loader still requires the legacy `dflash.n_target_layers` key to be present. Drafters published with the new metadata key naming (`dflash-draft.dflash.target_layer_ids` plus `n_target_features`) hit the path where the legacy key is missing and the loader fails. Concrete case: the published Q8 GGUF drafter for Qwen3.6-27B-DFlash. This change derives `n_target_layers` in two steps: 1. If `target_layer_ids` is present, use its length. 2. Otherwise, if `n_target_features` and `n_embd` are both present, use `n_target_features / n_embd` (with a sanity check that the division is exact). If neither is available, the loader still fails with the same honest error as before. The legacy key path is untouched. Validation (RTX 6000 Ada sm_89, Qwen3.6-27B Heretic Q4_K_M target, Q8 GGUF drafter via the new metadata): Loaded `SWA layers: 4/5`, decode 21.06 tok/s, no fallback chain errors during init. Verification vs existing community PRs: COMP-COMPL with Luce-Org#79 (merged 2026-05-03). Luce-Org#79 covered target loader and drafter fields generically. This PR is a small follow-up for the case where only the new metadata is present on the drafter side. Author: Javier Pazo <xabicasa@gmail.com>
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
…draft-loader-target-layers
dusterbloom
added a commit
to dusterbloom/lucebox-hub
that referenced
this pull request
May 12, 2026
…e-Org#119/Luce-Org#149 reorg Brings in HIP/Strix Halo backend (PRs Luce-Org#119, Luce-Org#149), dflash source-layout reorg (Luce-Org#138 — qwen35/, draft/, qwen3/ subdirs), GGUF draft loader fixes, daemon ubatch defaults, prefix cache + streaming tool-call fixes. Conflicts resolved: - dflash/CMakeLists.txt: take main's reorganized source paths; keep our gemma4_*.cpp entries; preserve the DFLASH27B_MIN_SM backwards- compat shim so gemma4_dflash_graph.cpp:621 keeps building under main's renamed _dflash27b_cuda_min_sm variable. - dflash/deps/llama.cpp: keep our submodule pointer (eb3676f40 on feature/tq3-kv-cache-clean). Main's c79573c9b lacks the TQ3 dispatcher fixes required for Gemma4 KV correctness; if useful upstream commits land there, they should be cherry-picked into our submodule branch separately. Verified: TQ3 64K MTP gamma=2 pflash post-merge: decode 10.58 tok/s, prefill 463 tok/s, accept 0.78 — matches pre-merge baseline (10.25 / 445 / 0.78) within noise.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(dflash): derive n_target_layers fallback in gguf_draft_loader
Follow-up to merged #79 ("read model params from GGUF at runtime,
support any qwen35 size"). #79 covers the target loader and the
common drafter fields, but the fallback chain in gguf_draft_loader
still requires the legacy
dflash.n_target_layerskey to bepresent.
Drafters published with the new metadata key naming
(
dflash-draft.dflash.target_layer_idsplusn_target_features) hit the path where the legacy key is missingand the loader fails. Concrete case: the published Q8 GGUF drafter
for Qwen3.6-27B-DFlash.
This change derives
n_target_layersin two steps:target_layer_idsis present, use its length.n_target_featuresandn_embdare bothpresent, use
n_target_features / n_embd(with a sanitycheck that the division is exact).
If neither is available, the loader still fails with the same
honest error as before. The legacy key path is untouched.
Validation (RTX 6000 Ada sm_89, Qwen3.6-27B Heretic Q4_K_M target,
Q8 GGUF drafter via the new metadata):
Loaded
SWA layers: 4/5, decode 21.06 tok/s, no fallback chainerrors during init.
Verification vs existing community PRs:
COMP-COMPL with #79 (merged 2026-05-03). #79 covered target
loader and drafter fields generically. This PR is a small
follow-up for the case where only the new metadata is present
on the drafter side.
Author: Javier Pazo xabicasa@gmail.com